Joint estimation from images

ABSTRACT

Techniques are disclosed for estimating poses from images. In one embodiment, a machine learning model, referred to herein as the “detector,” is trained to estimate animal poses from images in a bottom-up fashion. In particular, the detector may be trained using rendered images depicting animal body parts scattered over realistic backgrounds, as opposed to renderings of full animal bodies. In order to make appearances of the rendered body parts more realistic so that the detector can be trained to estimate poses from images of real animals, the body parts may be rendered using textures that are determined from a translation of rendered images of the animal into corresponding images with more realistic textures via adversarial learning. Three-dimensional poses may also be inferred from estimated joint locations using, e.g., inverse kinematics.

BACKGROUND Field

This disclosure provides techniques for estimating joints of animals andother articulated figures in images.

Description of the Related Art

Three-dimensional (3D) animal motions can be used to animate 3D virtualmodels of animals in movie production, digital puppeteering, and otherapplications. However, unlike humans whose motions may be captured viamarker-based tracking, animals do not comply well and are difficult totransport to confined areas. As a result, marker-based tracking ofanimals can be infeasible. Instead, animal motions are typically createdmanually via key-framing.

SUMMARY

One embodiment disclosed herein provides a computer-implemented methodfor identifying poses in images. The method generally includes renderinga plurality of images, where each of the plurality of images depictsdistinct body parts of at least one figure, and each of the distinctbody parts is associated with at least one joint location. The methodfurther includes training a machine learning model using, at least inpart, the plurality of images and the joint locations associated withthe distinct body parts in the plurality of images. In addition, themethod includes processing a received image using, at least in part, thetrained machine learning model which outputs indications of jointlocations in the received image.

Another embodiment provides a computer-implemented method fordetermining texture maps. The method generally includes converting,using adversarial learning, a plurality of rendered images that eachdepicts a respective figure to corresponding images that includedifferent textures than the rendered images. The method further includesextracting one or more texture maps based, at least in part, on (a)textures of the respective figures as depicted in the correspondingimages, and (b) pose and camera parameters used to render the renderedimages.

Another embodiment provides a computer-implemented method for extractingposes from images. The method generally includes receiving one or moreimages, each of the one or more images depicting a respective figure.The method further includes processing the one or more images using, atleast in part, a trained machine learning model which outputsindications of joint locations in the one or more images. In addition,the method includes inferring a respective skeleton for each image ofthe one or more images based, at least in part, on the joint locationsin the image.

Other embodiments include, without limitation, a computer-readablemedium that includes instructions that enable a processing unit toimplement one or more embodiments of the above methods, as well as asystem configured to implement one or more aspects of the above methods.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited aspects are attained andcan be understood in detail, a more particular description ofembodiments of the invention, briefly summarized above, may be had byreference to the appended drawings.

It is to be noted, however, that the appended drawings illustrate onlytypical embodiments of this invention and are therefore not to beconsidered limiting of its scope, for the invention may admit to otherequally effective embodiments.

FIG. 1 illustrates an example networked environment in which machinelearning models are trained and deployed for estimating poses in images.

FIG. 2 illustrates an approach for training and using a machine learningmodel to track joint locations, according to an embodiment.

FIG. 3 illustrates examples of a synthetic image and realistic versionsof images generated artificially using a neural network function,according to an embodiment.

FIG. 4 illustrates examples of an image depicting a mosaic of bodyparts, an image with manually labeled joint locations, and an image inwhich key points representing joints have been detected, according to anembodiment.

FIG. 5 illustrates a method for generating realistic animal textures andtraining a joint detector, according to an embodiment.

FIG. 6 illustrates a method for determining poses from images, accordingto an embodiment.

FIG. 7 illustrates a model generator computing system, according to anembodiment.

DETAILED DESCRIPTION

Embodiments presented herein provide techniques for estimating posesfrom images. Animals are used herein as a reference example, buttechniques disclosed herein may also be used to estimate poses of otherarticulated figures, such as humans, that are depicted in images.Animals do not wear clothes and have textures that vary less thanhumans, suggesting a synthetic annotated data set can be generated fortraining an animal pose detector by rendering a three-dimensional (3D)virtual model of the animal in various poses. However, large data setsof 3D animal motions do not exist, and it is difficult to createplausible animal poses by hand. A data set of animal motions may becreated by rendering only a small set of possible motions, but doing soinduces a bias toward a few poses such that a trained pose detectorcannot track motions outside of the initial data set. One embodimentinstead trains an animal pose detector to estimate animal poses fromimages in a bottom-up fashion so as to avoid pose distribution bias andinducing a quadruped structure. In such a case, the detector may betrained using rendered images depicting animal body parts scattered overrealistic backgrounds in different locations, orientations, and scales,as opposed to renderings of full animal bodies. In order to makeappearances of the rendered body parts more realistic so that thedetector can be trained to estimate poses from images of real animals,the body parts may be rendered using textures that are determined from atranslation of rendered images of the animal into corresponding imageswith more realistic textures via adversarial learning. Once trained, thedetector may be used to process images depicting the animal in order toestimate joint locations therein, together with confidence values forthe estimates. 3D poses may then be inferred from the joint locationsusing, e.g., inverse kinematics.

In the following, reference is made to embodiments of the invention.However, it should be noted that the invention is not limited tospecific described embodiments. Instead, any combination of thefollowing features and elements, whether related to differentembodiments or not, is contemplated to implement and practice theinvention. Furthermore, although embodiments of the invention mayachieve advantages over other possible solutions and/or over the priorart, whether or not a particular advantage is achieved by a givenembodiment is not limiting of the invention. Thus, the followingaspects, features, embodiments and advantages are merely illustrativeand are not considered elements or limitations of the appended claimsexcept where explicitly recited in a claim(s). Likewise, reference to“the invention” shall not be construed as a generalization of anyinventive subject matter disclosed herein and shall not be considered tobe an element or limitation of the appended claims except whereexplicitly recited in a claim(s).

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present invention are described below with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

Embodiments of the invention may be provided to end users through acloud computing infrastructure. Cloud computing generally refers to theprovision of scalable computing resources as a service over a network.More formally, cloud computing may be defined as a computing capabilitythat provides an abstraction between the computing resource and itsunderlying technical architecture (e.g., servers, storage, networks),enabling convenient, on-demand network access to a shared pool ofconfigurable computing resources that can be rapidly provisioned andreleased with minimal management effort or service provider interaction.Thus, cloud computing allows a user to access virtual computingresources (e.g., storage, data, applications, and even completevirtualized computing systems) in “the cloud,” without regard for theunderlying physical systems (or locations of those systems) used toprovide the computing resources.

Typically, cloud computing resources are provided to a user on apay-per-use basis, where users are charged only for the computingresources actually used (e.g. an amount of storage space consumed by auser or a number of virtualized systems instantiated by the user). Auser can access any of the resources that reside in the cloud at anytime, and from anywhere across the Internet. In the context of thepresent invention, a user may access applications (e.g., a jointestimation application) or related data available in the cloud. Forexample, a joint estimation application could execute on a computingsystem in the cloud to recognize joints and poses in images, and storethe recognized joints and poses at a storage location in the cloud.Doing so allows a user to access this information from any computingsystem attached to a network connected to the cloud (e.g., theInternet).

FIG. 1 illustrates an example networked environment 100 in which machinelearning models are trained and deployed for estimating poses in images,according to an embodiment. As shown, the networked environment 100includes a model generator 110, a training data repository 120, anapplication server 130, and a video repository 140 communicativelyconnected via a network 150.

The model generator 110 is generally representative of a computingsystem, such as the system discussed below with respect to FIG. 7. Asshown, the model generator 110 includes an image rendering application(“image renderer”) 112, a texture creation application (“texturecreator”) 114, and a model training application (“model trainer”) 116running therein. Although shown as being distinct, in some embodimentsthe image renderer 112, texture creator 114, and/or model trainer 116may be implemented as a single application. The image renderer 112 isconfigured to render images depicting 3D virtual scenes and, inparticular, images of synthetic animals that are rendered using texturemaps that do not appear realistic, which are also referred to herein as“synthetic images.” Well-known rendering techniques may be employed bythe image renderer 112, as discussed in greater detail below.

The model trainer 116 is configured to train machine learning models,which in one embodiment includes a model for translating syntheticimages of animals rendered by the image renderer 112 into versions ofthose images with realistic textures, from which the texture creator 114may generate realistic texture maps for the animals. In addition, themodel trainer 116 trains, using images depicting mosaics of body partsrendered with the realistic texture maps and associated joint locations,a detector machine learning model for predicting key points representingjoint locations in input images of the animal, as discussed in greaterdetail below. Training data, including the images depicting real animalsused to train the model that translates synthetic images to realisticversions of those images and the images depicting mosaics of body partsused to train the detector, may be stored in the training datarepository 120 (or elsewhere).

The application server 130 is generally representative of anothercomputing system, the components of which may be similar to those of themodel generator 110. As shown, the application server 130 includes ajoint estimation application 146 running therein. In one embodiment, thejoint estimation application 146 is configured to extract poses ofanimals depicted in input images using a trained detector to predict the2D locations of key points representing joints in the input images andinfer (e.g., via inverse kinematics) a 3D skeleton from the predictionjoint locations. As used herein, a “skeleton” refers to a hierarchicalset of interconnected bones that can be used to animate 3D virtualgeometries (i.e., 3D models). In particular, skeletons parameterizeposes, and a skeleton can be modified to deform an associated 3D model.The images input into the trained detector may include individual imagesand/or image frames of a video, which as shown are stored in andretrieved by the joint estimation application 146 from an imagerepository 140 (but may also be stored elsewhere).

FIG. 2 illustrates an approach for training and using a machine learningmodel to track joint locations, according to an embodiment. As shown,during synthetic image generation 100, the image renderer 112 performsrendering 205 with a 3D model 204 of an animal that is imported into a3D virtual scene, one or more animations 203 (e.g., walk and run cycles)used to pose the 3D model 204, blend shapes 202 used to deform the 3Dmodel 204 and introduce variability, and background images 201 depictingrealistic scenery. The rendering 205 composes the 3D model 204, posesfrom the animations 203, deformations caused by the blend shapes 202,and the background images 201 together to produce 2D images 206depicting a projected synthetic animal with realistic backgrounds. Inone embodiment, a virtual camera may be used to capture the 3D model 204from different viewpoints, such as various side views of the 3D model204, and backgrounds.

Panel A of FIG. 3 illustrates an example image 300 of a synthetic lion305 rendered without a background. In particular, the rendering is of a3D mesh, textured and parameterized by a skeleton via linear blendskinning. Illustratively, the synthetic lion 305 includes a texture thatdoes not appear realistic, as an unrealistic texture map (e.g., amanually created texture) was used during rendering of the image 300.The unrealistic texture map may be a rough approximation of a texturehaving a realistic appearance. It should be understood that an artistcan create a very realistic animal texture, but doing so may take a longtime and may still differ from reality given that texture is only partof a more complex computational rendering of the animal into an image.Another drawback of a manually created texture is variations in texturemust be created manually, which does not scale well. For a learned modelto generalize to different animals, the model needs to learn from manyvariations (i.e., the model needs to see many different textures duringtraining). Although the image 300 has been rendered without abackground, it should be understood that rendered images may generallyinclude backgrounds depicting realistic scenery.

Directly rendered images of synthetic animals (e.g., the image 300described above depicting the synthetic lion 305) may not appearrealistic compared to images depicting real animals (e.g., an imagedepicting a real-world lion). As a result, a detector trained toestimate joint locations using such images of a synthetic animal may notperform well when applied to estimate joint locations in imagesdepicting the real animal. That is, there is a reality gap problem. Oneembodiment solves such a problem by translating the synthetic images tocorresponding images that appear more realistic using adversariallearning, generating textures of the animal using the more realisticimages, and training a detector to predict joint locations using imagesthat depict body parts of the animal rendered with the generatedtextures and associated joint locations. Returning to FIG. 2, duringoffline generative adversarial network (GAN) training 210, the syntheticimages 206 and unlabeled animal images 213 are used to train a GAN 212including a neural network function, specifically a generative function,that may be used to artificially generate realistic versions of thesynthetic images 206 by modifying textures therein into more realistictextures that are similar to the textures in the unlabeled animal images213. The offline GAN training 210 uses a big data set (of syntheticimages and images depicting real animals) to obtain realistic animaltextures, which may help eliminate texture bias, while keeping theanimal poses in the synthetic images 206 unchanged.

Illustratively, the adversarial training trains a CycleGAN 212 using thesynthetic images 206 of an animal (e.g., a lion) as well as unlabeledreal-world images 213 of the same animal, and a generative function ofthe CycleGAN 212 is trained to generate realistic versions of thesynthetic images 206. A CycleGAN is a GAN architecture useful forlearning a mapping from one domain (e.g., that of the synthetic images206) to another (e.g., that of the unlabeled real-world images 213) withunpaired training data. In particular, the CycleGAN 212 may include agenerative function, also referred to herein as a “generator,” formapping input images in one domain to output images in another domainand a discriminative function, also referred to herein as a“discriminator,” that encourages the generator to create outputsindistinguishable from a desired data distribution domain, as well asanother generator and discriminator pair for the reverse mapping. Imagesgenerated using a GAN such as the CycleGAN 212 are also sometimesreferred to as “GANerated” images. The training of the CycleGAN 212 isloosely supervised, with no correspondence required between the 2Dprojected synthetic images 206 and the unlabeled images 213 of animalsthat are used as training data. Any suitable training algorithm may beemployed, such as the Adam optimization algorithm.

Panel B of FIG. 3 illustrates examples of images 310 and 320artificially generated using a neural network function. Illustratively,the images 310 and 320 each depict a lion with a realistic texture. Inone embodiment, the images 310 and 320 may be created by a generatorafter a CycleGAN including the generator is trained using synthetic andreal images (and the synthetic images and random noise are then inputinto the generator). In such a case, the generator of the CycleGAN maymodify the texture of animals depicted in the synthetic images (andbackgrounds of those images as the CycleGAN may be agnostic as to whatthe animals are) to appear more realistic.

Returning to FIG. 2, after the offline GAN training 210, the texturecreator 114 extracts GANerated texture maps 223 from the artificiallygenerated realistic versions of the synthetic images 206. It should beunderstood that the artificially generated realistic images depictrealistic looking animals for which pose and camera parameters areknown, as those pose and camera parameters were used to render thecorresponding synthetic images 206. That is, the offline GAN training210 process essentially creates a supervised setting with known poses,which was traditionally unavailable for animals that are less docilethan humans and cannot be made to assume certain poses in a controlledenvironment. The texture creator 114 makes use of the known pose andcamera parameters to generate the GANerated texture maps 223 by, e.g.,placing the synthetic animal over the artificially generated one foreach synthetic image and artificially generated realistic image pair andextracting the texture map from the artificially generated realisticimage. That is, assuming the 3D model of the animal is rendered from aside view with known pose and camera parameters, the location of the 3Danimal model in each artificially generated realistic image is knownand, as the 3D model is being projected from a known virtual cameraposition, the texture of the animal can be recovered from theartificially generated realistic image. Further, as each of theartificially generated realistic images may show only one side of theanimal (because the synthetic images were originally rendered from thesides of the animal), the texture creator 114 may mirror the animaltexture in the artificially generated realistic images, i.e., generatingthe GANerated texture maps 223 may include such mirroring. Any number oftexture maps (e.g., 3 to 100) may be generated in this manner. It shouldbe understood that multiple texture maps may provide variations intextures, reflecting the different appearances of animals in the realworld and permitting a detector to be trained to detect joint locationsin images depicting such animals.

Illustratively, the texture creator 114 blends the GANerated texturemaps 223 to generate 2D animal textures 221 that can be used tore-texture the 3D model of the animal. In one embodiment, the texturecreator 114 may employ linear blending, in which case linear weightedsums of randomly selected GANerated texture maps 223 may be determined.Optionally, manually-created texture maps 224 may also be blended withthe GANerated texture maps 223 if such manually-created texture maps 224are available (e.g., if a user manually created texture maps from theunlabeled animal images 213). Blending multiple a priori extractedtexture maps with each other and (optionally) with manually createdtexture maps allows diverse texture appearances to be created. It shouldbe understood that small artifacts in the texture maps 221 may notsignificantly affect a trained detector if the training data set issufficiently large and varied.

As shown, joint detector training 230 includes body part decompositionand data augmentation 231, during which the model trainer 116 (oranother application such as the image renderer 112) uses the 2D animaltextures 221 to render individual body parts of the animal withdifferent positions, orientations, scales, noisiness, illumination,blurriness, occlusions, warpings, etc. over a background. Body parts andassociated joints of a 3D animal model may be manually defined. In oneembodiment, for a 3D model of an animal having a skeleton embeddedtherein, with the 3D model being parameterized by the skeleton in linearblend skinning, the user may specify skeletal joints as belonging toparticular body parts (e.g., a knee joint belongs to a leg), andvertices of the 3D model geometry that are mapped to those joints inlinear blend skinning may be assigned to the body parts (i.e., thevertices mapped to joints associated with body parts are made into thebody parts).

The animal is decomposed and rendered as such body parts, rather than afull body, at random locations within the image, and data augmentationmay also be employed by, e.g., adding noise, rotating the body parts,scaling the body parts, warping the body parts, adding occlusions,changing lighting, etc. The result is a set of images depicting mosaicsof body parts, which are also labeled with joint locations that areknown based on the mapping of the 3D body parts and associated joints totwo dimensions. It should be understood that the images themselves arenot directly labeled; rather, as used herein, a “label” refers to adescription of a feature that should be predicted, in this case alocation of a joint which is associated with the image(s). The 3D modelof the animal may be placed in different poses to provide somevariability, and then cut into body parts for further variability.Rendering body parts, as opposed to full body poses, may help eliminatepose bias, as the detector can be trained to detect individual bodyparts in a manner that is agnostic to the overall pose of an animal, asopposed to detecting the animal in particular poses. The detector maythen be able to detect various motions such as jumping, laying on theground, getting up from the ground, etc. that the detector is notexplicitly trained to identify. Panel A of FIG. 4 shows an example image400 in which body parts of a lion (e.g., a leg 402) have been renderedusing the 2D animal textures 221 that appear realistic, and labeled 2Dkey points indicating the locations of joints (e.g., joint 404).

Returning to FIG. 2, images depicting mosaics of body parts and labeled2D key points 232 indicating joint locations therein are used astraining data to train a joint detector 235. Once trained, the jointdetector 235 takes images of an animal as input and predicts key pointsrepresenting joint locations in those images. In one embodiment, thejoint detector 235 may be a multi-stage hourglass deep neural networkthat predicts key points representing 2D joint locations from 2Dred-green-blue (RGB) images, together with a confidence value for eachpredicted joint location. An hourglass deep neural network is a neuralnetwork architecture with an hourglass shape that takes images as input,compresses the images to a bottleneck, and expands out from thebottleneck, in several stages (e.g., 5 or 6 stages). Other types ofmachine learning models, such as a convolutional pose machine, may beused in lieu of an hourglass deep neural network in alternativeembodiments. Given an input image, the trained joint detector 235 in oneembodiment may output a heat map for each joint indicating, at eachpixel location, a likelihood that the joint is located at that pixellocation. Although discussed herein primarily with respect to such heatmaps, the joint detector 235 may generally output any indication ofjoint locations and/or skeletons in input images.

As shown, the model trainer 116 may (optionally) perform fine tuning 234of the joint detector 235 using images with manually labeled jointlocations 234, assuming such manually labeled data is available. Panel Bof FIG. 4 shows an example image 410 depicting a lion and manuallylabeled joint locations, such as the location of joint 412. That is, adata set for the exact training purpose, including images depicting realanimals and manually labeled joint locations, may be used to fine tune(i.e., perform additional training of) the joint detector 235 after thejoint detector 235 has been trained using the images depicting mosaicsof body parts and associated 2D key points 232.

Returning to FIG. 2, after the joint detector 235 is trained, an image241 depicting a real animal may be input into the joint detector 235during joint extraction and pose estimation 240. In turn, the jointdetector 235 outputs key points indicating 2D joint locations 236extracted from the real animal image 241. Although one real animal image241 is shown for illustrative purposes, any number of images, such asthe image frames of a video, may be processing using the trained jointdetector 235.

In addition to extracting 2D joint locations, the joint estimationapplication 146 performs inference 242 using the joint locations 236 tobuild a 3D pose 243. In one embodiment, the 3D pose inference includesreconstructing a 3D pose by employing well-known optimization techniqueswhere the 2D key points act as constraints in the 3D pose optimizationprocess. Additionally, in the optimization process, kinematicconstraints, as well as pose priors may be used to further constrain thepose space. Panel C of FIG. 4 shows example key points (e.g., key point422) representing joints that have been detected in an image 420 of alion, as well as a skeleton inferred from such joints via inversekinematics.

FIG. 5 illustrates a method 500 for generating realistic animal texturesand training a joint detector, according to an embodiment. As shown, themethod 500 begins at step 510, where the image renderer 112 rendersscenes including a 3D model of an animal in multiple poses and variousbackgrounds, thereby generating synthetic images of the animal. Giventhe 3D model of the animal, animated poses, and (optionally) blendshapes for shape variation, the image renderer 112 may render imagesdepicting a set of animal poses on natural-looking backgrounds (e.g., alion on a savannah background). As described, the 3D model of the animalmay be rendered using a texture map that is unrealistic, and therendering may employ a virtual camera to capture the 3D model fromdifferent (side) viewpoints and backgrounds.

At step 520, the model trainer 116 trains a CycleGAN using the syntheticimages and real images of the animal as training data. Once trained,such a CycleGAN is capable of translating the appearances of syntheticand real animals (as well as backgrounds) in images. Although CycleGANis discussed herein as a reference example, other embodiments may useother machine learning models suitable for translating synthetic imagesto more realistic images.

At step 530, the texture creator 114 generates texture maps using thetrained CycleGAN. In one embodiment, the texture creator 114 inputs thesynthetic images rendered at step 510 (and random noise) into agenerator of the CycleGAN that maps the synthetic images to morerealistic versions of those images. Then, the texture creator 114extracts, based on known pose and camera parameters that were used togenerate the corresponding synthetic images from 3D virtual scenesincluding an animal model, texture maps for the animal. The extractedtexture maps may also be blended with each other and (optionally)manually-created texture maps, as described above. Multiple texture mapsmay be generated in such a manner to ensure variations in textures,reflecting the different appearances of animals in the real world.

At step 540, the model trainer 116 generates images that each depicts amosaic of body parts. Such images may include body parts of the animalmodel rendered with the texture maps generated at step 530 and randomlyscattered on realistic backgrounds. In addition, the model trainer 116may perform data augmentation, such as adding noise, rotating the bodyparts, scaling the body parts, warping the body parts, addingocclusions, changing lighting, etc.

At step 550, the model trainer 116 trains, using the images depictingmosaics of body parts and labeled locations of joints in those images, adetector that predicts joint locations. In one embodiment, the detectoris a deep neural network, and in particular the detector may be themulti-stage hourglass deep neural network described above that istrained to predict key points representing 2D joint locations andassociated confidence values. For example, the hourglass deep neuralnetwork may take as input pixel data of an image and output a heat mapfor each key point (joint). As described, a 3D skeleton may be inferredvia, e.g., inverse kinematics from 2D joint locations determined usingsuch a trained detector.

At step 560, the model trainer 116 (optionally) fine tunes the detectorusing images depicting the animal and manually labeled joint locationsin those images, if such manually labeled data is available.

FIG. 6 illustrates a method 600 for determining a pose from an image,according to an embodiment. As shown, the method 600 begins at step 610,where the joint estimation application 146 receives an image depictingan animal. Although discussed for simplicity with respect to a singleimage, steps of the method 600 may also be repeated for multiple images,such as the image frames of a video that depicts an animal, in order toextract skeletons (and an overall animation) from the images.

At step 620, the joint estimation application 146 processes the receivedimage using a trained detector to determine 2D joint locations therein.As described, the trained detector may, in one embodiment, be amulti-stage hourglass deep neural network that outputs respective heatmaps for each key point representing a joint of the animal. In such acase, the heat map associated with a joint may indicate, for each pixelin the image, a likelihood that the joint is located at that pixel. Toidentify the joint using such a heat map, the joint estimationapplication 146 may, e.g., determine pixels where the likelihoodindicated by the heat map exceeds a threshold value, and then take anaverage of the positions of the determined pixels.

At step 630, the joint estimation application infers a 3D skeleton basedon the joint locations using inverse kinematics. In alternativeembodiments, the skeleton may be determined in other ways. For example,the joint detector may itself be trained to predict skeletons, in whichcase the joint estimation application 146 may input a received imageinto such a detector to determine skeleton(s) therein.

FIG. 7 illustrates the model generator 110 computing system, accordingto an embodiment. Although the model generator 110 is shown forillustrative purposes, the application server 130 may generally includesimilar components and run the joint estimation application 146,described above. The model generator 110 and application server 130 mayalso be the same computing system in some embodiments. As shown, thesystem 110 includes, without limitation, a central processing unit (CPU)705, a network interface 715 connecting the system to a network 716, aninterconnect 717, a memory 720, and storage 730. The system 110 may alsoinclude an I/O device interface 710 connecting I/O devices 712 (e.g.,keyboard, display and mouse devices) to the system 110.

The CPU 705 retrieves and executes programming instructions stored inthe memory 720. Similarly, the CPU 705 stores and retrieves applicationdata residing in the memory 720. The interconnect 717 facilitatestransmission, such as of programming instructions and application data,between the CPU 705, I/O device interface 710, storage 730, networkinterface 715, and memory 720. CPU 705 is included to be representativeof a single CPU, multiple CPUs, a single CPU having multiple processingcores, one or more graphics processing units (GPUs), a combination ofthe above, and other types of processor(s). And the memory 720 isgenerally included to be representative of a random access memory. Thestorage 730 may be a disk drive storage device. Although shown as asingle unit, the storage 730 may be a combination of fixed and/orremovable storage devices, such as magnetic disk drives, flash drives,removable memory cards or optical storage, network attached storage(NAS), or a storage area-network (SAN). Further, system 110 is includedto be representative of a physical computing system as well as virtualmachine instance(s) hosted on underlying physical computing system(s).Further still, although shown as a single computing system, one ofordinary skill in the art will recognized that the components of thesystem 110 shown in FIG. 7 may be distributed across multiple computingsystems connected by a data communications network.

As shown, the memory 720 includes an operating system 721, the imagerenderer 112, the texture creator 114, and the model trainer 116. Theoperating system 721 may be, e.g., Linux® or Microsoft Windows®. Asdescribed, the image renderer 112 is configured to render imagesdepicting 3D virtual scenes, the model trainer 116 is configured totrain machine learning models, and the texture creator 114 is configuredto extract realistic textures of animals from, e.g., images generatedartificially using a neural network function such as the generator of atrained CycleGAN. In one embodiment, the image renderer 112 may renderinto synthetic images 3D virtual scenes including an animal in multipleposes and various backgrounds, after which the model trainer 116 maytrain a CycleGAN using the synthetic images and real images of theanimal as training data, the texture creator 114 may generate texturemaps using the trained CycleGAN, and the model trainer 116 may furthergenerate images depicting mosaics of body parts; train, using the imagesdepicting mosaics of body parts and labeled locations of joints in thoseimages, a detector that predicts joint locations; and (optionally) finetune the detector using images depicting the animal and manually labeledjoint locations in those images, according to the method 500 describedabove with respect to FIG. 5. Once the detector is trained, the jointestimation application 146 described above may determine poses in otherimages depicting the animal by processing those images using the traineddetector to extract joint locations therein and inferring skeletons fromthe joint locations via, e.g., inverse kinematics, according to themethod 600 described above.

Advantageously, techniques disclosed herein permit joints of animals andother articulated figures to be recognized in images. In particular,motions of animals can be learned from videos of real-world animalsusing techniques disclosed herein, in contrast to traditional animationsof animals that were hand-crafted. Such learned motions may then be usedto animate a 3D model of the animal in movie production, digitalpuppeteering, and other applications. For example, a running animationof a lion may be extracted from a video depicting a real lion running inthe savannah, and the running animation may then be used in acontrolling environment that permits a user to select and apply therunning animation to move a 3D lion model in a virtual scene. Bytranslating rendered synthetic images to more realistic images viaadversarial learning prior to training a detector, techniques disclosedherein can reduce or eliminate texture bias. Further, by training thedetector using rendered images depicting mosaics of body parts accordingto techniques disclosed herein, the detector may be capable of trackingmany different motions, as predictions by the detector are not biased bythe initial training data set.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder or out of order, depending upon the functionality involved. Itwill also be noted that each block of the block diagrams and/orflowchart illustration, and combinations of blocks in the block diagramsand/or flowchart illustration, can be implemented by special purposehardware-based systems that perform the specified functions or acts, orcombinations of special purpose hardware and computer instructions.

While the foregoing is directed to embodiments of the present invention,other and further embodiments of the invention may be devised withoutdeparting from the basic scope thereof, and the scope thereof isdetermined by the claims that follow.

What is claimed is:
 1. A computer-implemented method for identifyingposes in images, the method comprising: rendering a plurality of images,wherein: each of the plurality of images depicts distinct body parts ofat least one figure, and each of the distinct body parts is associatedwith at least one joint location; training a machine learning modelusing, at least in part, the plurality of images and the joint locationsassociated with the distinct body parts in the plurality of images; andprocessing a received image using, at least in part, the trained machinelearning model which outputs indications of joint locations in thereceived image.
 2. The method of claim 1, further comprising, inferringa skeleton based, at least in part, on the output joint locations. 3.The method of claim 2, wherein the skeleton is inferred using, at leastin part, inverse kinematics.
 4. The method of claim 1, wherein each ofthe depicted distinct body parts is a rendering of a portion of athree-dimensional (3D) virtual model using a texture map determined viaadversarial learning.
 5. The method of claim 4, wherein determining thetexture map via adversarial learning includes: training a cyclegenerative adversarial network (CycleGAN) using a plurality of renderedimages depicting respective figures and additional images depictingrespective figures in the real world; processing the plurality ofrendered images using the trained CycleGAN, thereby generatingcorresponding processed images; and generating the texture map using, atleast in part, the processed images.
 6. The method of claim 5, whereingenerating the texture map includes: extracting a plurality of texturemaps associated with the respective figures in the processed images; andblending at least two of the extracted texture maps.
 7. The method ofclaim 6, wherein the at least two of the extracted texture maps arefurther blended with at least one manually created texture map.
 8. Themethod of claim 1, wherein the machine learning model is a multi-stagehourglass deep neural network.
 9. The method of claim 1, wherein thedistinct body parts are rendered with at least one of random positions,orientations, scales, noisiness, illumination, blurriness, occlusions,or warpings.
 10. The method of claim 1, wherein the at least one figureincludes at least one animal.
 11. A computer-implemented method fordetermining texture maps, comprising: converting, using adversariallearning, a plurality of rendered images that each depicts a respectivefigure to corresponding images that include different textures than therendered images; and extracting one or more texture maps based, at leastin part, on (a) textures of the respective figures as depicted in thecorresponding images, and (b) pose and camera parameters used to renderthe rendered images.
 12. The method of claim 11, wherein: the figure isan animal rendered using, at least in part, a three-dimensional (3D)model and an initial texture map; and the converting the plurality ofrendered images includes: training a cycle generative adversarialnetwork (CycleGAN) using the rendered images and additional imagesdepicting the animal in real life, and processing the rendered imagesusing the trained CycleGAN.
 13. The method of claim 12, wherein thedifferent textures in the corresponding images include textures of theanimal that are more realistic than textures of the animal in therendered images.
 14. The method of claim 12, wherein the 3D model isrendered over a realistic background.
 15. The method of claim 12,further comprising: rendering a plurality of additional images, wherein:each of the additional images depicts distinct body parts of at leastone figure rendered using at least one of the one or more texture maps,and each of the distinct body parts is associated with at least onejoint location; training a machine learning model using, at least inpart, the additional images and the joint locations associated with thedistinct body parts in the additional images; and processing at leastone received image using, at least in part, the trained machine learningmodel which outputs indications of joint locations in the at least onereceived image.
 16. The method of claim 15, further comprising,inferring via inverse kinematics a respective skeleton for each image ofthe at least one received image based, at least in part, on the outputjoint locations in the image.
 17. The method of claim 11, wherein theextracted one or more texture maps includes a plurality of texture maps,and the method further comprises: blending at least two of the extractedtexture maps.
 18. The method of claim 17, wherein the at least two ofthe extracted texture maps are further blended with at least onemanually created texture map.
 19. A computer-implemented method forextracting poses from images, the method comprising: receiving one ormore images, each of the one or more images depicting a respectivefigure; processing the one or more images using, at least in part, atrained machine learning model which outputs indications of jointlocations in the one or more images; and inferring a respective skeletonfor each image of the one or more images based, at least in part, on thejoint locations in the image.
 20. The method of claim 19, wherein theinferring is performed via inverse kinematics.